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1  INTRODUCTION 

United  States  Coast  Guard  (USCG)  mission  responsibilities  have  grown  significantly  over  the  past  decade. 
Mission  growth  is  accompanied  by  a  commensurate  growth  in  system  and  tactical  complexity.  Both  types 
of  growth  are  influenced  by  changes  in  threat  behaviors,  requirements  for  internal  and  interagency 
coordination,  and  changes  to  doctrine  and  tactics.  USCG  platforms  and  systems  must  become  increasingly 
sophisticated  and  complex  to  meet  these  demands. 

Generally  speaking,  acquisition  risk  increases  with  system  complexity.  One  major  risk  involves  basing 
acquisition  decisions  on  potentially  erroneous  estimates  of  mission  performance  -  generally  overestimates 
due  to  engineering  optimism.  In  complex  systems,  performance  overestimation  often  occurs  due  to 
difficulties  in  understanding  how  system  components  and  the  mission  environment  interact  with  each  other 
and  how  these  interactions  influence  mission  performance.  Development  of  more  accurate  mission 
performance  estimates  can  help  prevent  unanticipated  performance  decrements  during  testing  and  fielding, 
and  thus  save  program  costs  and  prevent  the  schedule  delays  associated  with  late-stage  corrective  actions. 

Savvy  acquisition  Program  Managers  (PMs)  who  want  to  mitigate  these  risks  will  take  every  reasonable 
action  to  develop  high-quality  information  about  their  systems.  Human  performance  modeling  can  be  a  key 
component  in  these  efforts,  especially  when  coupled  with  other,  more-traditional  modeling,  simulation,  and 
analysis  approaches  available  to  the  PM. 

2  MODELING  AND  SIMULATION:  AN  INFORMATION  SOURCE  FOR  THE 
ACQUISITION  PM 

Modeling  and  simulation  (M&S)  provides  a  cost-effective  means  for  developing  information  and  knowledge 
about  system  concepts  and  estimating  their  mission  performance  potential.  M&S-based  analysis  can  be 
applied  to  inform  nearly  every  step  in  the  acquisition  process:  from  pre-acquisition  mission  analysis;  to 
analysis  of  as-built  systems;  to  supporting  the  development  of  Tactics,  Techniques,  and  Procedures  (TTP). 
M&S  techniques  are  particularly  well-suited  for  developing  and  exploring  system  trade-offs  and 
requirement  specifications,  using  approaches  similar  to  that  depicted  in  Figure  1.  First,  alternative  system 
concepts  are  proposed  in  terms  of  functionality  and  the  capability  levels  of  subsystems  and  components 
(Fig.  1,  top  left).  The  system  concepts  are  then  captured  in  a  constructive  simulation  of  a  representative 
mission  in  the  context  of  a  real-world  scenario  (Fig.  1,  bottom  left).  The  mission  simulation  environment 
facilitates  analyses  influenced  by  the  same  dynamic  demands  that  the  system  must  serve  in  the  real  world. 
This  includes  key  factors  relating  to  demand  types,  frequencies,  and  magnitudes  that  are  difficult  to  account 
for  in  isolated  analyses  of  individual  components.  Scenario  outcomes  are  measured  objectively  in  terms  of 
predicted  mission  performance  along  with  other  supporting  metrics  (Fig.  1,  top  right).  Component 
performance  levels  linked  to  objective,  quantified  overall  system  performance  goals  then  become  the  basis 
of  system  requirement  specifications  (Fig.  1,  bottom  right). 
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System  Requirements 

•  The  unmanned  aerial  vehicle 
(UAV)  shall  have  an  endurance 
of  36  hours 

•  The  UAV  shall  have  a  cruise 
speed  of  100  kts 

•The  radar  on  the  UAV  shall 
have  a  range  of  35  NM 


Figure  1.  An  illustration  of  how  M&S  is  used  to  develop  system  requirements. 

In  practice,  numerous  alternatives  are  proposed  and  evaluated  using  this  process.  When  compared  to 
development  and  testing  of  physical  prototypes,  M&S-based  alternatives  analyses  are  accomplished  more 
quickly,  executed  at  far  less  cost,  and  remain  more  flexible.  This  translates  to  an  ability  to  examine  and 
refine  many  variations  in  types  and  levels  of  system  capabilities  under  a  range  of  missions  and  conditions. 
Furthermore,  M&S-based  analysis  can  include  tactical  situations  from  the  riskiest  part  of  the  operational 
spectrum  without  risking  the  lives  and  health  of  personnel.  Such  analysis  is  often  necessary  to  develop  a 
proper  understanding  of  system  performance  and  limitations  in  extreme  conditions.  Finally,  and  perhaps 
most  importantly,  some  of  the  system  concepts  formulated  and  tested  using  the  M&S  process  may  fail  to 
deliver  the  required  levels  of  mission  performance.  Such  failures  in  early  analysis  activities  are  okay  - 
especially  when  they  are  discovered  before  sunk  costs  begin  to  accumulate.  Knowing  both  what  works  and 
what  does  not  work  in  a  system  concept  helps  to  define  the  capability  boundaries  of  an  effective  solution. 

In  many  ways  the  process  of  proposing,  testing,  and  refining  alternative  system  concepts  is  as  important  as 
selecting  the  one  to  implement.  Taken  together  across  the  complete  set  of  system  alternatives,  the  PM 
benefits  from  an  extensive,  objective  knowledge-base  and  can  therefore  make  well-informed  decisions. 

Like  any  analytic  method,  the  quality  and  accuracy  of  the  results  obtained  with  M&S  and  the  success  of  the 
decision-making  it  supports  are  dependent  upon  the  quality  and  accuracy  of  the  implemented  system  models 
and  mission  environment  representations.  It  is  critical  that  the  system  model  represent  all  of  the  system 
components  and  elements  that  contribute  to  mission  performance  and  further  capture  the  salient  aspects  of 
all  of  the  entities,  factors,  and  attributes  that  stress  and  drive  the  system.  Simulations  that  fail  to  do  this  can 
result  in  inaccurate  predictions  of  a  system’s  performance,  opening  the  door  to  poor  acquisition  decisions. 
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3  HUMAN  PERFORMANCE  MODELS  ENABLE  TOTAL  SYSTEM  ANALYSIS 

Recent  research  conducted  by  the  USCG  Research  &  Development  Center  (RDC)  has  shown  that 
incorporation  of  detailed  representations  of  personnel  roles  and  actions  within  system  models  can  greatly 
improve  the  quality  and  accuracy  of  system  performance  estimates.  This  finding  is  important  because 
nearly  all  of  the  constructive  simulations  used  to  develop  system  requirements  include  only  highly- 
aggregated  (i.e.,  summary-level)  representations  of  personnel  performance.  Personnel  are  a  critical 
component  of  most  systems;  they  make  the  most  difficult  and  significant  decisions  and  they  perform 
activities  too  complex  to  automate.  In  short,  personnel  are  primary  contributors  to  the  mission  and, 
therefore,  need  to  be  represented  in  any  simulation  that  estimates  total  system  performance. 

There  is  a  capability  for  incorporating  portrayals  of  personnel  into  current  models  and  simulations  used  to 
represent  and  test  systems.  It  is  called  human  performance  modeling  (HPM).  RDC  research  has 
demonstrated  how  HPM  can  be  integrated  with  simulations  used  to  test  system  concepts.  Figure  2 
illustrates  the  conceptual  architecture  employed  and  shows  how  operator-focused  system  requirements  can 
be  generated  along  with  requirements  for  the  physical  system. 


Alternative  1  Alternative  2  1 

Endurance 

24  hrs 

36  hrs 

Cruise  Speed 

120  kts 

1 00  kts 

Radar  Range 

40  NM 

35  NM 

Operator  Tasks  Interface  1  Interface  2 


Prioritize  1 5  sec  5  sec 
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Evaluate  Image  30  sec  30  sec 
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Alternative 


Perceptual 
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Measures  of  Effectiveness 

%  Thrts  %  Vessels  %  Vessels  Alternative  1 

IDed  Classified  IDed 

I  Alternative  2 


System  Requirements 

The  UAV  shall  have  a  cmise  speed  of  1 00 
kts 

The  operator  shall  be  able  to  employ  a 
sensor  within  60  seconds 


Figure  2.  Illustration  of  how  HPMs  can  be  integrated  with  current  acquisition  simulations. 

Essentially,  HPMs  provide  computational  representations  of  how  people  perform  work  and  other  activities. 
The  models  represent  human  tasks,  processes,  information-processing,  decision-making,  control  actions, 
communications,  and  other  required  behaviors  in  terms  of  human  abilities  such  as  perception,  cognition,  and 
psychomotor  actions.  Just  as  models  can  be  adapted  to  represent  the  functionality  and  capability  of  physical 
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components  associated  with  a  system  concept,  an  HPM  can  be  shaped  to  represent  how  an  operator  would 
perform  within  a  system  concept  based  on  anticipated  roles  and  system  interfaces.  Individual  or  collective 
HPM  elements  can  be  adapted  to  exchange  data  and  commands  with  constructive  simulation  model 
elements  to  create  an  integrated  simulation  representation.  As  shown  in  Figure  2,  the  constructive  model 
passes  perceptual  information  (such  as  the  data  on  a  cockpit  display)  to  the  HPM,  where  it  is  “perceived”  by 
the  simulated  operator.  The  HPM  “decides”  what  action  to  take  and  sends  data  back  to  the  system  model  to 
implement  the  action.  The  result  is  an  integrated,  dynamic  representation  of  the  total  system  -  the  hardware, 
software,  mission  environment,  and  people. 

4  USE  OF  HPM  CAN  IMPROVE  SYSTEM  PERFORMANCE  ESTIMATES 

RDC  researchers  were  interested  in  knowing  whether  integrated  HPM-constructive  simulation  environments 
would  produce  mission  performance  results  that  are  different  from  those  produced  by  traditional,  “stand¬ 
alone”  constructive  simulations.  The  accredited  Coast  Guard  Tactical  Modeling  Environment  (CGTME) 
was  used  to  construct  a  simulation  of  an  unmanned  aerial  vehicle  (UAV)  executing  a  surveillance  mission  in 
the  Florida  Straits.  In  this  mission,  the  UAV  is  searching  for  two  types  of  threat  vessels  -  small,  high-speed 
boats  (“Go-Fasts”)  and  even  smaller,  slower-moving  vessels  (“Chug-Chugs”)  -  carrying  illegal  drugs  and/or 
migrants.  Legitimate  vessel  traffic  typical  of  the  Florida  Straits  environment  was  modeled  as  well,  and 
included  freighters,  tankers,  cruise  ships,  commercial  fishing  boats,  and  recreational  motor  vessels.  The 
density  of  legitimate  traffic  was  varied  to  represent  typical  and  high  traffic  densities  (150  and  250  vessels, 
respectively).  The  threat  density  was  held  constant  at  three  Go-Fasts  plus  three  Chug-Chugs. 

The  scenario’s  UAV  is  controlled  remotely  from  a  ground  station  by  a  crew  consisting  of  a  pilot  and  a 
sensor  system  operator  (SSO).  The  mission  profile  required  the  UAV  to  fly  a  pre-determined  flight  path. 
Consequently,  the  pilot  had  a  minimal  role.  The  SSO  was  the  key  player,  using  the  UAV’s  sensors  to  search 
for  threats  (primary  mission)  while  attempting  to  detect,  classify,  and  identify  all  other  traffic  (secondary 
mission).  The  UAV  was  equipped  with  multi-mode  radar  capable  of  both  vessel  detection  and 
classification;  an  electro-optical  (EO)  sensor  that  supported  vessel  identification;  and  an  Automatic 
Identification  System  (AIS)  receiver,  which  could  detect  and  track  all  traffic  with  AIS  transceivers.  The 
CGTME  constructive  model  used  simple  modeling  algorithms  for  performing  detections,  classifications, 
and  identifications,  using  time  delays  to  represent  aggregated  SSO  actions.  These  time  delays  were 
established  based  on  subject  matter  expert  (SME)  estimates  of  the  overall  time  required  to  accomplish  an 
aggregated  process  (e.g.,  the  total  amount  of  time  it  takes  to  classify  a  vessel). 

To  create  the  integrated  simulation,  a  HPM  was  developed  -  using  the  IMproved  Performance  Research 
INtegration  Tool  (IMPRINT)  -  that  implemented  the  details  of  the  full  range  of  actions  that  an  SSO  would 
perform.  The  HPM  was  developed  using  inputs  from  USCG  sensor  and  surveillance  mission  SMEs  -  but  in 
this  case,  the  SMEs  were  asked  to  examine  each  individual  task  element  within  the  overall  process.  The 
HPM  was  connected  to  the  CGTME  through  a  custom  software  link.  In  this  integrated  mode,  the 
constructive  model’s  detection,  classification,  and  identification  algorithms  were  disabled.  Instead,  the 
IMPRINT  SSO  model  received  data  from  the  CGTME  UAV  model  that  represented  the  information  a  real 
operator  would  receive  from  the  sensor  system  interfaces.  The  SSO  model  processed  this  information  to 
make  decisions  about  which  vessel  to  look  at  next,  what  sensor  to  use  and  how  to  use  it,  and  the  status  and 
classification  of  the  vessel  currently  being  presented  on  a  sensor  display.  The  SSO  model  then  passed  data 
back  to  the  CGTME  UAV  model  as  required  to  implement  SSO  model  decisions  (e.g.,  select  and  point  a 
sensor,  record  an  identified  target,  etc.)  and  to  score  the  results. 
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Ten  data  collection  runs  were  conducted  for  both  the  stand-alone  and  integrated  simulations  at  both  traffic 
density  levels  (scenario  sizes).  Two  key  bottom-line  metrics  were  evaluated:  (1)  threat  identification 
percentage,  and  (2)  non-threat  classification  percentage.  The  results  are  summarized  in  Table  1. 

As  noted  earlier,  there  were  six  threat  (smuggling)  vessels  in  each  scenario.  The  results  show  that  the  stand¬ 
alone  CGTME  model  always  identified  all  six  of  the  threat  vessels  but  the  integrated  HPM-CGTME  model 
found  only  four  or  five  of  these  threats  (67%  -  80%  in  Table  1).  It  is  also  important  to  note  that  the 
percentage  of  threats  found  by  the  HPM-CGTME  model  decreased  as  overall  traffic  density  increased. 

From  a  mission  performance  standpoint,  the  stand-alone  CGTME  and  the  integrated  HPM-CGTME  made 
significantly  different  predictions  of  system  performance. 


Table  1.  Experiment  results  summary. 


Simulation  Mode 

Performance  Metrics 

Percentage  of  Threats 
Identified  (of  6) 

Percentage  of  Non-Threat 
Vessels  Classified 

Scenario  Size: 

150  Vessels 

250  Vessels 

150  Vessels 

250  Vessels 

Stand-alone  CGTME 

100% 

100% 

62% 

62% 

Integrated  HPM-CGTME 

80% 

67% 

28% 

14% 

For  the  secondary  mission  objective  (classification  of  legitimate  traffic),  the  differences  in  the  results  are 
even  more  dramatic.  In  the  stand-alone  CGTME  simulation,  almost  two-thirds  of  the  non-threat  vessels 
were  identified;  this  was  true  for  both  levels  of  traffic  density  (scenario  size).  But  in  the  integrated  HPM- 
CGTME  simulation,  the  results  vary  with  traffic  density:  only  28  percent  of  vessels  were  classified  in  the 
150-vessel  condition,  and  that  decreased  to  14  percent  in  the  high-density  (250-vessel)  condition.  The 
HPM-CGTME  results  are  consistent  with  human  factors  research,  which  shows  that  as  the  number  of  targets 
and  amount  of  workload  increase,  human  vigilance  and  performance  decrease. 

Why  do  we  get  such  remarkable  differences  between  two  methods  of  simulating  the  same  system?  The 
answer  lies  primarily  in  the  different  levels  of  detail  at  which  the  detection,  classification,  and  identification 
events  are  represented.  In  the  stand-alone  CGTME  simulation,  these  events  are  each  single-step  processes. 
For  example,  once  a  target  vessel  is  selected  for  classification  and  comes  within  classification  range, 
classification  success  is  assured,  and  incurs  only  a  time  delay  to  be  completed  (approximately  60  seconds). 
In  the  integrated  simulation,  the  HPM  is  responsible  for  classification  -  a  process  that  is  represented  as  a 
multi-step  sequence  of  tasks  (e.g.,  enter  radar  coordinates,  slew  EO,  step  through  magnification  levels  until 
vessel  features  are  visible,  etc.)  that  varies  based  on  target  range  and  vessel  type.  This  more-realistic 
process,  as  represented  in  the  HPM,  can  take  from  one  to  several  minutes.  Because,  on  average,  the  HPM- 
based  simulation  of  operator  tasks  takes  more  time  to  evaluate  each  vessel  than  does  stand-alone  CGTME,  it 
(and,  presumably,  a  real  operator)  simply  cannot  process  as  many  vessels  during  the  scenario. 
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5  CONCLUSIONS 

Imagine  you  are  a  PM  responsible  for  procuring  a  UAV  like  the  one  described  earlier.  You  are  presented 
with  two  alternatives  for  assessing  the  UAV  concept. 

•  The  first  alternative  is  to  use  a  stand-alone  constructive  model  to  predict  threat  identification 
performance.  Based  on  its  (overly-optimistic)  prediction  that  the  system  will  identify  100%  of 
threats,  you  make  a  significant  investment  to  acquire  an  initial  UAV  prototype  for  testing.  Then 
during  testing,  you  discover  the  bitter  truth:  human  operators  cannot  perform  their  tasks  fast  enough 
to  support  the  promised  level  of  performance.  Your  time,  money,  and  energies  are  now  likely  to  be 
consumed  in  attempting  to  mitigate  this  shortfall. 

•  Your  other  (preferred)  alternative  is  to  take  the  extra  step  early  in  the  process  to  incorporate  HPM 
and  integrated  simulations  in  your  analysis  work.  You  set  more  realistic  goals  and  expectations  for 
your  program,  or  seek  out  technology  solutions  that  can  lead  to  better  overall  performance. 

Bottom  Line:  When  using  M&S  to  assess  total  system  performance,  it  is  important  to  incorporate  operator 
representations  into  models  of  the  physical  system.  This  helps  ensure  proper  representation  of  all  system 
components  that  drive  mission  performance,  resulting  in  more  accurate  and  reliable  predictions  of  total 
system  performance. 

6  ADDITIONAL  NOTES 

The  authors  wish  to  point  out  that  there  are  a  number  of  different  types  of  simulations  that  can  be  used  in  the 
acquisition  process  for  a  wide  range  of  analyses.  The  focus  of  this  paper  is  solely  on  constructive 
simulations  that  are  used  to  predict  total  system  performance.  CGTME  is  an  excellent  M&S  tool,  and  can 
be  employed  in  stand-alone  mode  to  serve  a  wide  variety  of  analyses.  Key  examples  include  analysis  aimed 
at  assessing  alternative  physical  system  configurations,  especially  those  where  the  competing  alternatives 
imply  similar  duties  for  system  operators.  We  do  not  mean  to  imply  that  HPM  must  be  incorporated  into  all 
CGTME-based  tests  or  with  other  USCG  simulations.  HPM  is  just  one  tool  in  the  M&S  practitioner’s 
toolkit.  As  with  any  tool,  it  must  be  used  appropriately  to  obtain  cost-effective  analysis  results. 

7  FOR  MORE  INFORMATION 

Readers  interested  in  learning  more  about  the  experiment,  the  related  models  and  simulations,  or  how  to 
incorporate  integrated  modeling  into  their  own  projects  are  urged  to  contact  the  RDC. 

Dr.  Anita  Rothblum  (860-271-2847,  Anita.M.Rothblum@uscg.mil)  leads  human  factors  and  HPM  work 
at  the  RDC  and  was  the  CG  project  manager  on  this  effort. 

Mr.  Kevin  Downer  (860-271-2654,  Kevin.F.Downer@uscg.mil)  is  a  senior  operations  research  analyst 
at  the  RDC  and  is  familiar  with  a  range  of  M&S  tools. 
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